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DETAILED ACTION 
Response to Arguments 

Applicant's arguments with respect to claims 1-22 have been considered but are 
moot in view of the new ground(s) of rejection. 

Claim Objections 

Claim 21 is objected to because of the following informalities: The preamble of 
claim 21 recites, "A multimedia computing, comprising:" It is assumed that the preamble 
should recite, "A multimedia computing system, comprising:" 

Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1-5, 8-11, 14, 16, and 17 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Flamini et al. (US 2004/0100486) in view of Griesau et al. (US 
6,507,306). 

Regarding claim 1, Flamini describes a multimedia computing system, 
comprising: a graphics application that generates computer graphics in response to 
user input; an input control application that controls at least one of video input source in 
response to user input; and a video processing engine, or pixel processing engine, that 
receives input from at least one of the video input sources and receives computer 
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graphics from the graphics application, the video processing engine, or pixel processing 
engine, alpha blending the computer graphics with the input and outputting the blended 
data in substantially real time. For example, Flamini describes a video processing 
system that comprises a user interface of control elements that overlay video content on 
a display paragraph 0053. The video processing software is stored on a computer- 
readable medium that is executed by a computer paragraph 0054, which inherently 
means that the video processing system includes a real time operating system that 
supports the software program. The software program, or graphics application, 
generates computer graphics in response to user input; in this case the program 
displays menus, thumbnail images, and tools upon request from the user paragraphs 
0070 and 0071. The input control application is a basic remote control and it controls 
the video processing engine, which receives video input from the video input source and 
computer graphics from the graphics application paragraph 0070. The video processing 
engine then alpha blends the computer graphics with the background content and 
outputs the blended images in real time paragraphs 0070 and 0072. 

Flamini doesn't describe an input control application that directly controls a 
plurality of dissimilar video playback input sources in response to user input. 

However, Griesau describes an input control application that directly controls a 
plurality of dissimilar video playback input sources in response to user input. 
Specifically, Griesau describes a universal remote control unit 10 (see Figure 1) for 
controlling a plurality of devices 20. The universal remote control unit 10 can directly 
control a videocassette recorder, a digital video disk player, or a laser-disk player, which 
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are considered dissimilar video playback input sources (column 3 lines 10-29). It would 
have been obvious to one of ordinary skill in the art at the time of the invention to 
include in Flamini the system and method of directly controlling a plurality of video 
playback devices from a single remote unit, as taught by Griesau, in order to allow a 
user to easily control any one of the various input devices that may be coupled to the 
photo information appliance 202 (see paragraphs 0076-0078 of Flamini) from a single 
remote control. The advantage of using a single remote control to directly control a 
plurality of input devices is that it simplifies the experience for the user. A user can 
quickly and easily switch among and control video input devices all from the same user 
interface, which means a user will have an easier time working with the program and 
will be more inclined to use it. 

Concerning claim 2, Flamini includes a system comprising an input/output 
interface that communicates with a user input device. For example, Flamini describes 
that a user can interact with the video processing software by using a standard remote 
control that is configured to communicate with the application paragraph 0070. 

With regard to claim 3, Flamini discloses a system in which the I/O interface 
communicates with the mass storage device. For example, Flamini offers Figure 4, 
which illustrates photo information appliance 202 coupled to mass storage device 206. 
The system communicates with the mass storage device when it needs to access 
stored images or other stored data paragraph 0075. 

As to claim 4, Flamini describes a system in which the video processing engine 
receives input from the mass storage device and overlays the computer graphics on the 
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input received from the mass storage device in real time. For example t Flamini 
discloses that the mass storage device connected to the photo information appliance 
202 (of Figure 4) is capable of storing a large number of digital images, such as photos 
paragraph 0075. Flamini further describes that the user is allowed to manipulate the 
selected photos, accessed from the mass storage device, in real time paragraph 0070. 

Regarding claim 5, Flamini describes a system in which the I/O interface is a 
USB interface. For example, Flamini includes that an input device 208 (of Figure 4) can 
be coupled to the photo information appliance 208 to provide images to the photo 
information appliance. Flamini further describes that the possible input devices include 
devices that are connected via USB paragraph 0076. 

Concerning claim 8, Flamini includes a system in which the plurality of video 
input sources have a plurality of different protocols. For example, Flamini describes that 
the video input device can be connected via USB, 1394 Firewire, or other 
communication protocol paragraph 0076. 

Concerning claim 9, Flamini describes a video processing engine that blends 
computer graphics and video and can accept a number of different communication 
protocols, including USB and 1394 (Firewire) paragraph 0076. Flamini doesn't describe 
that the protocols also include one of RS-232, RS-422, Control-M, LANC, and infrared. 

However, it would have been obvious to one of ordinary skill in the art at the time 
of the invention to include in Flamini the Recommended Standard serial protocols, such 
as RS-232 and RS-422, in order to ensure that the photo information appliance is 
compatible with devices that aren't equipped with USB or 1394 (Firewire) serial 
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communication ports but are equipped with standard 9-pin serial ports that follow RS- 
232 protocol. 

With regard to claims 10 and 11, Flamini describes a system in which the 
plurality of video input sources generate video streams having a plurality of different 
formats, and in which the different formats comprise at least one of YU V, RGB, S-video, 
composite, VGA, and DVI. For example, Flamini discloses that the system can accept 
digital video signals in many well-known formats, such as BNC composite, serial digital, 
parallel digital, RGB, or consumer digital video paragraph 0077. 

As to claim 14, Flamini describes a system in which the video processing engine 
outputs the blended data to a plurality of output devices. For example, Flamini 
describes a video processing engine that uses alpha blending to blend graphics and 
video paragraph 0072, and further describes the processed images can be output to 
any number and type of output devices including a laser printer, Zip™ drive, CD, DVD, 
the Web, and email paragraph 0079. 

Regarding claim 16, Flamini discloses a system in which blended data is stored 
as an image file. For example, Flamini describes that when the photo information 
appliance has completed the processing of the digital image a user can output the 
image to locations such as a Zip™ drive, CD, or DVD paragraph 0079. Flamini further 
describes that the mass storage devices save the data in the form of digital images 
paragraph 0075. 

Concerning claim 17, Flamini includes a video processing system that is 
portable. For example, Flamini offers Figure 4, which illustrates the photo information 
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appliance 202 potentially coupled to a television set 204, output devices 210, input 
devices 208, and peripheral devices 206. These devices are all optional attachments, 
but alone the photo information appliance is simply a small computer system, which is 
considered portable. 

Claims 6, 7, 21, and 22 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Flamini et al. (US 2004/0100486) in view of Griesau et al. (US 
6,507,306) further in view of White et al. (US 6,909,438) and further in view of Fang (US 
2004/0257369). 

As to claims 6 and 7 Flamini in view of Griesau describes a video processing 
engine comprising a software program that blends computer graphics and video, as 
discussed in the rejection of claim 1 . Flamini in view of Griesau doesn't describe that 
the video processing engine comprises a dedicated logic circuit or that the dedicated 
logic circuit comprises a field programmable gate array (FPGA). 

However, White describes a system of blending two images, such as a video 
source and a still image, which can be implemented wherein some or all of the software 
is replaced by a dedicated logic circuit, such as an FPGA column 2 lines 61-66 and 
column 4 lines 1-13. It would have been obvious to one of ordinary skill in the art at the 
time of the invention to include in Flamini in view of Griesau the implementation option 
of using a field programmable gate array as taught by White in order to create a more 
efficient video processing device that has minimal delay in completing a set number of 
tasks while using a minimal amount of hardware. Flamini in view of Griesau and further 
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in view of White doesn't describe that the video processing engine also comprises at 
least one encoder, at least one video decoder, and a video buffer. 

However, Fang describes a system that blends graphics and video and does 
comprise at least one encoder, at least one video decoder, and a video buffer. For 
example, Fang offers Figure 1, which includes an MPEG, or video, decoder 1 10, an 
analog NTSC/PAL decoder 200, an NTSC/PAL encoder 170, a video frame buffer 120, 
and a graphic and video frame buffer 150. It would have been obvious to one of 
ordinary skill in the art at the time of the invention to include in Flamini, Griesau, and 
White the encoder, video decoder, and video buffer as taught by Fang in order to 
provide a means of decoding incoming data, buffering the video data until the graphics 
processor is ready to blend the video with graphics, and encoding the video so it is 
compatible with a display device that requires NTSC/PAL encoded data. The 
advantage of these added components is that the decoders will allow the system to be 
compatible to a wider range of inputs, the buffers will allow the system to process 
information with less delay, and the encoder will allow the system to output information 
to a wider range of display devices. 

Concerning claim 21, Flamini describes a multimedia computing system, 
comprising: a central processing unit (CPU); an input/output (I/O) interface that receives 
user input via a wireless user input device; a graphics application that generates 
computer graphics in response to user input received from the user input device, the 
application interfacing with the CPU via an operating system; a device remote control 
application that controls a plurality of different video input devices in response to user 
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input received from the user input device; and a video processing engine, or pixel 
processing engine, that receives input from at least one of the video input sources and 
receives computer graphics from the graphics application, the pixel processing engine 
alpha blending the computer graphics with the input and outputting the blended data in 
real time. For example, Flamini describes a software program that blends computer 
graphics and a video signal. The software program is described as being stored on a 
computer-readable medium arranged to cause a host computer system to execute 
operations paragraphs 0053 and 0054. The host computer system is considered a real 
time video processing system, comprising: a CPU and a real time operating system that 
communicates with the CPU. The host computer system also has an I/O interface that 
receives user input via a wireless user input device paragraph 0070, in this case a 
standard remote control, and a graphics application that generates computer graphics in 
response to user input received from the user input device paragraphs 0071 and 0072, 
in this case drop down menus and tools to modify images. The device remote control 
application controls a plurality of different video input devices in response to user input 
received from the user input device paragraphs 0070 and 0076, in this case the user 
has control over the video input device currently connected to the photo information 
appliance, and the photo information appliance is compatible with a plurality of input 
devices. Finally, Flamini describes that the real time embedded video processing 
system accepts input from a video source and overlays this input with computer 
graphics by alpha blending the computer graphics and video input together to make a 
composite image paragraphs 0070 and 0072. 
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Flamini doesn't describe an input control application that directly controls a 
plurality of dissimilar video playback input sources in response to user input. 

However, Griesau describes an input control application that directly controls a 
plurality of dissimilar video playback input sources in response to user input. 
Specifically, Griesau describes a universal remote control unit 10 (see Figure 1) for 
controlling a plurality of devices 20. The universal remote control unit 10 can directly 
control a videocassette recorder, a digital video disk player, or a laser-disk player, which 
are considered dissimilar video playback input sources (column 3 lines 10-29). It would 
have been obvious to one of ordinary skill in the art at the time of the invention to 
include in Flamini the system and method of directly controlling a plurality of video 
playback devices from a single remote unit, as taught by Griesau, in order to allow a 
user to easily control any one of the various input devices that may be coupled to the 
photo information appliance 202 (see paragraphs 0076-0078 of Flamini) from a single 
remote control. The advantage of using a single remote control to directly control a 
plurality of input devices is that it simplifies the experience for the user. A user can 
quickly and easily switch among and control video input devices all from the same user 
interface, which means a user will have an easier time working with the program and 
will be more inclined to use it. 

Flamini in view of Griesau doesn't teach that the video processing engine 
comprises a field programmable gate array (FPGA) comprising a CPU interface and a 
pixel processing engine that receives input from at least one of the video input sources 
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and receives computer graphics from the graphics application via the CPU and CPU 
interface. 

However, White describes a video processing engine comprises a field 
programmable gate array (FPGA) comprising a CPU interface and a pixel processing 
engine that receives input from at least one of the video input sources and receives 
computer graphics from the graphics application via the CPU and CPU interface. For 
example, White discloses a system that receives a video signal and a computer 
graphics signal and blends the two signals using alpha blending column 2 lines 61-67 
and column 3 lines 1-16. White further discloses that the system can be accomplished 
using a combination of software and hardware, where the software is stored on one or 
more processor readable storage media and the dedicated hardware can be an FPGA 
column 4 lines 1-13. If the system comprises both a CPU and dedicated hardware such 
as an FPGA then it is inherent that the system also comprises a CPU interface to allow 
the CPU and dedicated hardware to communicate. It would have been obvious to one 
of ordinary skill in the art at the time of the invention to include in Flamini in view of 
Griesau the FPGA as taught by White in order to create a more efficient video 
processing device that has minimal delay in completing a set number of tasks while 
using a minimal amount of hardware. 

Flamini in view of Griesau and further in view of White doesn't teach that the 
FPGA also comprises a video buffer, at least one video decoder, at least one video 
encoder, and a video buffer controller, and that the pixel processing engine receives 
input from at least one of the video input sources via at least one of the video decoders 
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and outputs the final blended computer graphics and video data in real time to at least 
one of the video encoders. 

However, Fang describes a system that takes in video data and computer 
graphics data and outputs an image consisting of blended computer graphics and video 
paragraph 0007. Fang also includes Figure 1, which illustrates video frame buffer 120, 
Analog NTSC/PAL video decoder 200, and NTSC/PAL video encoder 170. Graphics 
processor 130 combines computer graphics and video; it takes in the video through the 
NTSC/PAL decoder or MPEG decoder and outputs the final blended computer graphics 
and video data in real time through the NTSC/PAL encoder as required. It is inherent 
that the system described by Fang includes video buffer controllers that control the 
movement of data into and out of the video frame buffers, as the buffers would be 
useless without logic that dictates when saving data to each buffer is allowed and when 
accessing data within each buffer is allowed. It would have been obvious to one of 
ordinary skill in the art at the time of the invention to include in Flamini in view of 
Griesau and further in view of White the video encoder, video decoder, video buffer 
controller, and video buffer as taught by Fang in order to provide a means of decoding 
incoming data, buffering the video data until the graphics processor is ready to blend 
the video with graphics, and encoding the video so it is compatible with a display device 
that requires NTSC/PAL encoded data. The advantage of these added components is 
that the decoders will allow the system to be compatible to a wider range of inputs, the 
buffers will allow the system to process information with less delay, and the encoder will 
allow the system to output information to a wider range of display devices. 
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Concerning claim 22, Flamini includes a video processing system that is 
portable. For example, Flamini offers Figure 4, which illustrates the photo information 
appliance 202 potentially coupled to a television set 204, output devices 210, input 
devices 208, and peripheral devices 206. These devices are all optional attachments, 
but alone the photo information appliance is simply a small computer system, which is 
considered portable. 

Claims 12 and 13 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Flamini et al. (US 2004/0100486) in view of Griesau et al. (US 6,507,306), as 
applied to claim 1 above, and further in view of Lin (US 5,936,683). 

Regarding claims 12 and 13 Flamini in view of Griesau describes that the photo 
information appliance 202 of Figure 4 can accept a plurality of digital and analog video 
signals. Flamini further describes that the photo information appliance can accept an 
analog signal from an analog television and that the image processor uses an analog- 
to-digital converter to convert the analog signal into a discrete series of digitally 
encoded numbers to create a data word suitable for digital processing paragraph 0078. 
Flamini in view of Griesau doesn't describe translating the video from a YUV signal 
format to an RGB signal format by digitally color mapping the YUV format into an RGB 
format. 

However, Lin describes translating the YUV signal format into an RGB format 
using digital color mapping. For example, Lin describes a process of using color look- 
up tables, which is considered color mapping, to convert YUV signals into RGB signals 
column 2 lines 33-67 and column 3 lines 1-20. It would have been obvious one of 
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ordinary skill in the art at the time of the invention to include in Flamini in view of 
Griesau the conversion from YUV format to RGB format as taught by Lin in order to use 
a simple and proven method of converting a YUV signal, from an analog television for 
instance, into an RGB signal so Flamini's photo information appliance can accept and 
digitally process signals without putting unnecessary strain on the photo information 
appliance's central processor. 

Claim 15 is rejected under 35 U.S.C. 103(a) as being unpatentable over Flamini 
et al. (US 2004/0100486) in view of Griesau et al. (US 6,507,306) and further in view of 
Shen et al. (US 6,401,059). 

Concerning claim 15, Flamini in view of Griesau describes using an input device 
comprising a client application that communicates with a host application within the 
system, as described in the rejection of claim 1 . For example, Flamini includes that a 
user can interact with the software program using a standard remote control paragraph 
0070. Flamini in view of Griesau doesn't describe that the remote control comprises a 
personal digital assistant (PDA). 

However, Shen describes a method and system for using a PDA as a remote 
control column 2 lines 21-37. It would have been obvious to one of ordinary skill in the 
art at the time of the invention to include in Flamini in view of Griesau the PDA that acts 
as a personal remote control as taught by Shen in order to give the user one remote 
control that is capable of a variety of functions such as giving program schedule 
information for television shows, or controlling a variety of appliances such as Flamini's 
photo information appliance, a television, DVD player, VCR, and sound system. The 
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multi-functional PDA remote simplifies a user's interaction with many devices, which 
makes the devices easier to manage and the addition of new devices, such as a photo 
information appliance, less of a burden and therefore more appealing. 

Conclusion 

The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. Cavallaro et al. (US 6,864,886) describes a telestration system 
that allows a user to graphically annotate live or stored video, Leapman et al. (US 
2003/0198008) describes a computer system that has a wireless touch screen display, 
and Cormack et al. (US 2005/0097451) describes annotating media content with user- 
specified information. 

Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Dan Washburn whose telephone number is (571) 272- 
5551. The examiner can normally be reached on Monday through Friday 8:30 a.m. to 
5:00 p.m.. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka Chauhan can be reached on (571) 272-7782. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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